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Abstract Body 


Background / Context: 

This study addresses the effectiveness of a nationally used core reading program that reflects the 
research-based practices recommended by the National Reading Panel. This and other similar 
programs are increasingly used to prevent reading difficulties and ensure that all children are 
reading at or above grade level by the end of third grade. Converging evidence from two decades 
of research suggests that with appropriate instruction, nearly all students can become competent 
readers (Denton & Mathes, 2003; Lyon, Fletcher, Fuchs, & Chhabra, 2006; Mathes & Denton, 
2002; Snow, Burns, & Griffin, 1998). Yet statistics indicate that 65% of fourth grade students 
and 64% of eighth grade students fail to reach proficient-level reading scores (National Center 
for Educational Statistics, 2013). Recent initiatives emphasize the critical role of early reading 
instruction in preventing reading difficulties, recognizing that students who do not learn to read 
well by third grade are less likely to build vocabulary and interact with a wide variety of texts 
(Good, Simmons, & Kame’enui, 2001). Such failure can have a long-term impact on children’s 
self-confidence, motivation to learn, performance in school, and success in life (Harris & Sipay, 
1990; Juel, 1988; Stanovich, 1986, 2000), and reading difficulties are the most common reason 
for referral into special education (Donovan & Cross, 2002). Despite these concerns, however, 
only a few replicable beginning reading programs reviewed by the What Works Clearinghouse 
(2007) have more than a small amount of evidence suggesting that they have potentially positive 
effects: Success for All; Voyager; Reading Recovery; and Ladders to Literacy 1 . 

The Open Court Reading (OCR) program, published by SRA/McGraw-Hill and widely used 
since the 1960s, offers a phonics-based K-6 curriculum that shows promise for preventing 
reading difficulties. According to market research, OCR is among the top reading series 
(Educational Market Research, 2002). To date, a total of 2,917 districts and over 8,600 schools 
have adopted the OCR program across all 50 states and Washington DC (SRA/McGraw-Hill, 
2008)". Lindings from independent non-experimental evaluations suggest that, in comparison to 
other reading curricula, OCR is associated with significantly better reading outcomes and may be 
particularly effective with low-performing students (Edsource, 2006; McRae, 2002; Skindrud & 
Gersten, 2006). In addition, a recent cluster randomized efficacy trial documented the impact of 
the OCR program on reading achievement in grades 1 through 5 in 5 schools across the country. 
Results revealed one-year classroom-level impacts of treatment assignment of approximately one 
fifth of a standard deviation on the Comprehensive Test of Basic Skills (Borman, Dowling, & 
Schneck, 2007). Despite the program’s widespread use and promising research findings, OCR 
has not been evaluated rigorously on a large scale as part of an objective, third-party evaluation. 

Purpose / Objective / Research Question / Focus of Study: 

Effective early reading instruction is critical for preventing later reading difficulties. With two- 
thirds of 4th grade students failing to achieve proficiency in reading on the National Assessment 
of Educational Progress in 2013, the need to implement effective early reading programs is 

1 These programs were rated as having a “medium to large” amount of evidence which requires at least two studies 
that meet the WWC evidence screen with 2 schools and a total sample size of at least 350 students or 14 classrooms 
across the studies. “Potentially positive effects” is evidence of a positive effect in a domain with no overriding 
contrary evidence. 

2 The latest version of the Open Court Reading program is called Imagine It! 
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greatly apparent. The Open Court Reading (OCR) program published by SRA/McGraw-Hill and 
widely used for almost 40 years is a phonics-based core-reading program for students in 
kindergarten to 6th grade that incorporates many of the instructional practices related to 
phonemic awareness, phonics, fluency, vocabulary, and text comprehension recommended by 
the National Reading Panel of 2000. In this study, an independent research team evaluated the 
effectiveness of the OCR program in a large national sample of elementary schools at scale, 
across diverse school populations and conditions, and with no more support than schools would 
have access to if they had selected OCR as their early reading curriculum apart from 
participation in a research project. 

The results of this study contribute to an understanding of whether OCR is effective in 
promoting reading proficiency in the elementary grades when implemented “at scale” with 
typical “real world” levels of support. The study was designed to address the following research 
questions: 

• Overall Impacts. Does school-level assignment to the OCR curriculum intervention 
produce stronger effects on reading achievement than assignment to the “business-as- 
usual” control condition? 

• Impacts by Subgroups. Is there significant variation in the outcomes of OCR or do the 
effects reliably replicate across student subgroups, the sampled classrooms/teachers, 
schools, and districts? 

• Fidelity of Implementation. To what extent was the intervention delivered as the 
curriculum developers indicated it should be implemented? Was there significant 
variation in implementation fidelity of OCR among the classrooms/teachers, schools, and 
districts? In what ways were OCR students’ experiences similar or different to those of 
students in the control condition? 

• Proximal Outcomes as Moderators of Impacts. Is there a significant relationship 
between proximal student and teacher outcomes, such as fidelity of implementation or 
student motivation/engagement, and student reading achievement outcomes and does this 
relationship vary by classrooms/teachers, schools, and districts? 

Setting: 

The study was conducted in a sample of 49 elementary schools (kindergarten through 5th grade) 
in 7 districts across the country. 

Population / Participants / Subjects: 

The study participants include approximately 4,500 elementary school students and 1,200 
teachers per year. 

Intervention / Program / Practice: 

The OCR program is widely used and incorporates the instructional practices recommended in 
the 2000 report of the National Reading Panel related to phonemic awareness, phonics, fluency, 
vocabulary, and text comprehension. The OCR curriculum includes student materials, teacher 
manuals, diagnostic and assessment tools, and test preparation practice guides. The program 
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includes a 2- or 3 -day summer workshop to train teachers on program implementation and on- 
going support by OCR reading consultants throughout the school year. In all grades 
(kindergarten through 5th grade), the instructional format is a three-part lesson with specific 
instruction on vocabulary, comprehension strategies, and comprehension skills. Both informal 
and fonnal assessments are used to monitor progress and inform instruction. 

Research Design: 

The study design is a multi-site cluster-randomized trial in which 49 elementary schools from 7 
districts across the country were randomized to training and delivery of the OCR curriculum 
(treatment group) or to delivery of the standard reading instruction for the school (control group) 
blocking at the district level. 

Data Collection and Analysis: 

Data from teachers and students in two cohorts (grades K&3 and grades 1&4) were gathered 
over two school years. This paper is presenting findings from the first year. The pre- and post- 
test outcomes were assessed in the fall and spring using the Group Reading Assessment and 
Diagnostic Evaluation (GRADE) and Student Motivation Form (SMF). Fidelity of 
implementation was captured using classroom observations, interviews, and surveys with 
teachers and other key staff (e.g., curriculum trainers). 

The main intent-to-treat (ITT) impact analyses uses a three-level model with school level fall 
pre-test scores on the GRADE as a covariate and spring post-test scores as the dependent 
variable, nested within schools, which in turn were nested within districts. Additional subgroup 
(moderator) analyses will be used to investigate the effects of the OCR program as a function of 
student baseline characteristics (e.g., age/grade, gender, baseline math proficiency, student 
engagement), teacher/classroom characteristics (e.g., class size, fidelity of implementation), and 
school characteristics (e.g., geographic region or locale). Student, teacher, and school 
characteristics will also be examined as potential mediators of the effects. 

Findings / Results: 

Table 1 provides school level characteristics of students in the study schools at baseline. The 
analytical sample is comprised of 4,488 students in 49 study schools in grades K and 3 with valid 
scores on the Spring GRADE assessment used to measure students’ reading proficiency. These 
data were pulled from the 2011-2012 CCD files and collected from schools. The sample is a 
heterogeneous group of students based on race/ethnicity and socioeconomic status. There is an 
even split by sex and grade. This table also provides a test of equivalence for students enrolled in 
schools randomized to the OCR treatment condition and students enrolled in business as usual 
schools. At baseline, there were no significant differences between the demographic 
characteristics of schools in the treatment and control conditions. 


Insert Table 1 here 
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Preliminary results from the overall ITT impact analyses are presented in Table 2. The Intraclass 
Correlation (ICC) of student reading achievement was 0.086 for schools and 0.029 for the district 
level. The pretest scores were equivalent across treatment ( X = 92.40, SD = 5.92) and control 
schools ( X = 92.77, SD = 4.14). 

The three level multilevel model (student, school, and district) includes grand-centered, school- 
mean GRADE pretest scores and an indicator for treatment condition. Both predictor variables of 
pretest and treatment condition are included in level 2 of the model. 

Level 1 : 

Yijkl — Pojk ^ijkl 

Level 2 : 

Po jk = Yooj + Yoij (M eanPretest) jk + y ^Treatment) jk + p ojk 
Level 3 : 

Koo = n ooo + C ooo/c 

Y oi = n oio + (oio/c 

Y 02 = n 020 + Co20fc 


The multilevel model results indicate that the school mean pretest is predictive of the posttest 
achievement. A one standard deviation increase in school pretest score is associated with a 0.89 
increase in outcome scores. The effect of assignment to the treatment condition is positive, but it 
is not statistically significant. 


Insert Table 2 here 


This preliminary analysis indicates that the OCR program does not have a statistically significant 
impact on students’ reading achievement compared to the business as usual curriculum in place 
in control schools after the first year of implementation. The results indicate that variation in the 
treatment effect by district was greater than one standard deviation. This possible effect will be 
explored further in additional analyses focusing on subgroup variation and differences in fidelity 
of implementation. 


Insert Figure 1 here 


Conclusions: 

Early preliminary results indicate no statistically significant impacts of OCR after one year when 
implemented at scale relative to business as usual reading programs in a sample of 49 schools 
Year one findings will be further explored by assessing fidelity of implementation and subgroup 
impacts as well as potential mediation pathways in the full paper. 
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Appendix B. Tables and Figures 

Table 1. School Characteristics for Study Schools and by Treatment Assignment at Baseline 




All Schools 

OCR (Imagine It!) 
Treatment Schools 

Business as 
Usual Control 
Schools 

p-value 

Title 1 Eligible (%) A 


74% 

65% 

83% 

0.151 

Schoolwide Title I Eligible (%) A 


74% 

65% 

83% 

0.151 

Students Eligible for Free or 
Reduced Price Meals (%) A 


63% 

61% 

65% 

0.587 

Student Enrollment (Average) 

Grade K 

70 

69 

72 

0.745 


Grade 3 

66 

64 

68 

0.580 

Student Gender (%) 

Male 

52% 

52% 

52% 

0.842 


Female 

48% 

48% 

48% 

0.842 

Student Race/Ethnicity (%) 

White 

66% 

67% 

64% 

0.658 


Non-Hi spanic black 

13% 

12% 

15% 

0.672 


Hispanic 

12% 

12% 

12% 

0.889 


Other 

9% 

9% 

9% 

0.986 

ELL (%) 


12% 

11% 

12% 

0.805 

Special Education (%) 


10% 

11% 

10% 

0.423 

Schools 


49 

25 

24 


Students enrolled at Baseline 


6,804 

3,350 

3,454 



Note A: Data for these variables was pulled from the most recent CCD files available - 201 1/2012. However, one study school was 
first operational in the 2013/2014 school year. This school is not represented in these data. 
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Table 2. Multilevel Model Estimates for Impact of OCR (Imagine It!) on Student Reading Achievement 


Fixed effects model 

Coefficient 

Standard 

Error 

Degrees of 
freedom 

t-ratio 

p-value 

Intercept 

98.139 

0.505 

9.072 

194.196 

0.000 

GRADE Pretest (grand- 
mean, school mean 
score) 

0.886 

0.060 

50.780 

14.793 

0.000 

Treatment 

0.642 

0.556 

38.155 

1.155 

0.255 


Random effects 

Variance 

component 

Standard 

deviation 

School Level 

1.315 

1.730 

District Level 

0.781 

0.609 
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Predicted Posttest Scores, GRADE 


Figure 1. Predicted Posttest GRADE Scores from 3 Level Model Across 7 Districts 
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